Apparatus, systems, and methods for low power computational imaging

ABSTRACT

The present application discloses a computing device that can provide a low-power, highly capable computing platform for computational imaging. The computing device can include one or more processing units, for example one or more vector processors and one or more hardware accelerators, an intelligent memory fabric, a peripheral device, and a power management module. The computing device can communicate with external devices, such as one or more image sensors, an accelerometer, a gyroscope, or any other suitable sensor devices.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims benefit of the earlier priority date of U.S.Provisional Patent Application No. 62/030,913, entitled “LOW POWERCOMPUTATIONAL IMAGING COMPUTING DEVICE,” filed on Jul. 30, 2014. Thisapplication also claims priority as a continuation-in-part of U.S.patent application Ser. No. 14/082,396, entitled “APPARATUS, SYSTEMS,AND METHODS FOR PROVIDING COMPUTATIONAL IMAGING PIPELINE,” filed on Nov.18, 2013, which claims priority to the Romanian Patent Application OSIMRegistratura A/00812, entitled “APPARATUS, SYSTEMS, AND METHODS FORPROVIDING CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGING PIPELINE,”filed on Nov. 6, 2013, and to the U.K. Patent Application No.GB1314263.3, entitled “CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGINGPIPELINE,” filed on Aug. 8, 2013. Each one of the applications is herebyincorporated by reference herein in its entirety.

FIELD OF THE APPLICATION

This present application relates generally to providing a low powercomputational imaging computing device.

BACKGROUND

Computational imaging is a new imaging paradigm that is capable ofproviding unprecedented user-experience and information based on imagesand videos. For example, computational imaging can process images and/orvideos to provide a depth map of a scene, provide a panoramic view of ascene, extract faces from images and/or videos, extract text, features,and metadata from images and/or videos, and even provide automatedvisual awareness capabilities based on object and scene recognitionfeatures.

While computational imaging can provide interesting capabilities, it hasnot been widely adopted. The slow adoption of computational imaging canbe attributed to the fact that computational imaging comes withfundamental data processing challenges. Oftentimes, image resolution andvideo frame rates are high. Therefore, computational imaging generallyrequires hundreds of gigaflops of computational resources, which may bedifficult to obtain using regular computer processors, especially wherethat performance has to be sustainable and backed up by high memorybandwidth at low power dissipation. Furthermore, computational imagingis generally sensitive to latency. Because users are unlikely to waitseveral minutes for a camera to recognize an object, computationalimaging cameras are generally designed to process images and videosquickly, which further burdens the computational requirement ofcomputational imaging.

Unfortunately, it is difficult to implement computational imagingtechniques in customized hardware. As the field of computational imagingis in its relative infancy, implementation techniques are in constantflux. Therefore, it is difficult to customize computational imagingentirely in hardware as changes to implementation techniques wouldrequire redesigning the entire hardware. Accordingly, it is generallydesirable to provide a flexible hardware architecture and a flexiblehardware infrastructure.

At the same time, the demand for such video and image processing iscoming to a large extent from portable electronic devices, for exampletablet computers and mobile devices, where power consumption is a keyconsideration. As a result, there is a general need for a flexiblecomputational imaging infrastructure that can operate even under aconstrained power budget.

SUMMARY

In accordance with the disclosed subject matter, systems and methods areprovided for providing low power computational imaging.

Disclosed subject matter includes a computing device. The computingdevice can include a plurality of vector processors, wherein one of theplurality of vector processors is configured to execute an instructionthat operates on a first array of values. The computing device can alsoinclude a hardware accelerator configured to perform a filteringoperation on a second array of values. The computing device can alsoinclude a memory fabric comprising a plurality of memory slices and aninterconnect system having a first interface and a second interface,wherein the first interface is configured to couple the plurality ofvector processors to the plurality of memory slices and wherein thesecond interface is configured to couple the hardware accelerator to theplurality of memory slices. In addition, the computing device caninclude a host processor configured to cause the memory fabric toprovide the first array of values to the one of the plurality of vectorprocessors via the first interface and to provide the second array ofvalues to the hardware accelerator via the second interface, therebyenabling the one of the plurality of vector processors to process thefirst array of values in accordance with the instruction and enablingthe hardware accelerator to process the second array of values inaccordance with the filtering operation.

In some embodiments, the computing device can include a plurality ofpower islands each comprising at least one power domain, wherein a firstof the plurality of power islands is coupled to a first supply voltageto provide the first supply voltage to one of the plurality of vectorprocessors, and wherein a second of the plurality of power islands iscoupled to a second supply voltage to provide the second supply voltageto the hardware accelerator.

In some embodiments, the computing device can include a power managementmodule configured to provide an enable signal to a switch that couplesthe first of the plurality of power islands to the first supply voltage,thereby placing the one of the plurality of vector processors into anactive mode.

In some embodiments, the one of the plurality of vector processors cancomprise a logic circuit region for processing the first array of valuesand local memory for storing at least a subset of the first array ofvalues, and the power management module can be configured to cause thefirst supply voltage to be provided to the logic circuit region and tocause a third supply voltage to be provided to the local memory tocontrol a power consumption of the logic circuit region and the localmemory independently.

In some embodiments, the power management module can be configured toturn off the switch to disconnect the first of the plurality of powerislands from the first supply voltage, thereby placing the one of theplurality of vector processors into a low-power mode.

In some embodiments, the power management module can comprise a validsignal generator configured to generate a valid signal, indicating atime instance at which circuit blocks in the first of the plurality ofpower islands are ready to process input data, wherein the valid signalgenerator comprises a daisy chain of switches that provides the firstsupply voltage to the circuit blocks in the first of the plurality ofpower islands.

In some embodiments, the computing device can include a peripheraldevice coupled to a plurality of input/output (I/O) pins, wherein theperipheral device is configured to provide a communication channelbetween at least one of the plurality of vector processors and anexternal device.

In some embodiments, the peripheral device can be within a power islandthat is always powered on.

In some embodiments, the peripheral device can be configured to monitorsignals from the external device to detect an event to which one of theplurality of vector processors should respond to, and when theperipheral device detects the event, cause the power management moduleto place the one of the plurality of vector processors into the activemode.

In some embodiments, the peripheral device can comprise an emulationmodule that is configured to cause the peripheral device to emulate afunctionality of a plurality of standard protocol interfaces via acommon set of the I/O pins.

In some embodiments, the peripheral device can be coupled to adifferential pair of I/O pins, and the peripheral device is configuredto change a polarity of the differential pair based on a polaritycontrol signal.

In some embodiments, the differential pair of I/O pins can comprise adifferential pair of Mobile Industry Processor Interface (MIPI) lanes.

In some embodiments, the peripheral device can comprise a bypass bufferthat is configured to perform a bypass between an input I/O pin and anoutput I/O pin, thereby providing a communication channel between theinput I/O pin and the output I/O pin without placing the one of thevector processors in an active mode.

Disclosed subject matter includes a method. The method can includeproviding a memory fabric comprising a plurality of memory slices and aninterconnect system having a first interface and a second interface. Themethod can also include coupling, using the first interface, theplurality of memory slices and a plurality of vector processors, andcoupling, using the second interface, the plurality of memory slices anda hardware accelerator. The method can further include providing, by thememory fabric, a first array of values to one of the plurality of vectorprocessors via the first interface and providing a second array ofvalues to the hardware accelerator via the second interface, executing,at the one of a plurality of vector processors, an instruction thatoperates on the first array of values, and performing, by the hardwareaccelerator, a filtering operation on the second array of values.

In some embodiments, the method can include providing a first supplyvoltage to one of the plurality of vector processors, and providing asecond supply voltage to the hardware accelerator, wherein the one ofthe plurality of vector processors and the hardware accelerator areassociated with a first power island and a second power island,respectively.

In some embodiments, the method can include providing, by a powermanagement module, an enable signal to a switch that couples the firstpower island to the first supply voltage, thereby placing the one of theplurality of vector processors into an active mode.

In some embodiments, the method can include generating a valid signal,indicating a time instance at which circuit blocks in the first powerisland are ready to process input data, using a daisy chain of switchesthat provides the first supply voltage to the circuit blocks in the oneof the plurality of vector processors.

In some embodiments, the method can include providing a peripheraldevice coupled to a plurality of input/output (I/O) pins, wherein theperipheral device is associated with a power island that is alwayspowered on.

In some embodiments, the method can include monitoring signals from anexternal device to detect an event to which the one of the plurality ofvector processors should respond to, and causing the power managementmodule to place the one of the plurality of vector processors into theactive mode.

In some embodiments, the method can include emulating, by the peripheraldevice, a functionality of a plurality of standard protocol interfacesvia a common set of the I/O pins.

In some embodiments, the peripheral device is coupled to a differentialpair of I/O pins, and the method further comprises changing a polarityof the differential pair based on a polarity control signal.

In some embodiments, the method can include performing a bypass betweenan input I/O pin and an output I/O pin using a bypass buffer, therebyproviding a communication channel between the input I/O pin and theoutput I/O pin without placing the one of the vector processors in anactive mode.

Disclosed subject matter includes an electronic device. The electronicdevice can include a plurality of vector processors, wherein one of theplurality of vector processors is configured to execute an instructionthat operates on a first array of values. The electronic device can alsoinclude a hardware accelerator comprising a programmable datapathpipeline that is programmed using configuration information receivedfrom a software module, wherein the programmable datapath pipeline isconfigured to perform a filtering operation on a second array of valuesin accordance with the configuration information. The electronic devicecan also include a memory fabric comprising a plurality of memoryslices. The electronic device can further include a host processorconfigured to cause the memory fabric to provide the first array ofvalues to the one of the plurality of vector processors and to providethe second array of values to the hardware accelerator, thereby enablingthe one of the plurality of vector processors to process the first arrayof values in accordance with the instruction and enabling the hardwareaccelerator to process the second array of values in accordance with theconfiguration information.

In some embodiments, the hardware accelerator can include an outputbuffer for receiving a scan-line of an image processed by theprogrammable datapath pipeline, and a pipeline stall controllerconfigured to stall an operation of the programmable datapath pipelinewhen the output buffer is full.

In some embodiments, the hardware accelerator can include a plurality offunctional units that are chained together to perform the filteringoperation.

In some embodiments, an order in which the plurality of functional unitsis chained together is determined using the configuration informationreceived from the software module.

In some embodiments, an output of a first of the plurality of functionalunits is provided to a buffer in a memory fabric, and an input of asecond of the plurality of functional units is received from the buffer.

In some embodiments, the hardware accelerator can include a depth mapclient that is configured to receive depth information that isindicative of a depth of an object represented by a pixel in thescan-line of the image.

In some embodiments, the hardware accelerator can include a depth mapmodule that is configured to process the depth information to match aresolution of the depth information to a resolution of the scan-line ofthe image.

In some embodiments, the depth map module is configured totime-synchronize the depth information to the scan-line of the image.

In some embodiments, the memory fabric can include a mutual-exclusion(mutex) controller that is configured to monitor a status of anexclusive access request requesting an exclusive access to a sharedresource by one of the vector processors, and when the one of the vectorprocessors receives an exclusive access to the shared resource, send anacknowledgement message to the one of the vector processors, indicatingthat the one of the vector processors has the exclusive access to theshared resource.

In some embodiments, the memory fabric can include a plurality ofbuffers, wherein a first of the plurality of buffers is associated witha first of the vector processors, and wherein a second of the vectorprocessors is configured to send data to the first of the vectorprocessor by storing the data in the first of the plurality of buffers.

In some embodiments, the memory fabric can be configured to dynamicallymodify a capacity of the first of the plurality of buffers based on anamount of data transferred to the first of the vector processors.

In some embodiments, the memory fabric can be configured to dynamicallyassociate two or more of the plurality of buffers to the first of thevector processors based on an amount of data transferred to the first ofthe vector processors.

In some embodiments, the plurality of buffers can be a part of one ofthe plurality of memory slices in the memory fabric.

In some embodiments, the memory fabric can be configured to store stateinformation of one of the vector processors when the one of the vectorprocessors enters a low-power mode.

In some embodiments, the state information is stored in a static randomaccess memory in the memory fabric.

In some embodiments, the memory fabric can include a direct memoryaccess (DMA) controller, wherein the DMA controller comprises anoperation list indicating an order in which DMA operations are to beperformed.

In some embodiments, the DMA controller can be configured to perform asubset of the DMA operations in the operation list based on an enablebuffer, wherein the enable buffer includes a plurality of bits, whereinone of the plurality of bits is associated with one of the DMAoperations, and a value of the one of the plurality of bits isindicative of whether the one of the DMA operations is to be performedby the DMA controller.

Disclosed subject matter includes a method. The method can includeproviding, by a memory fabric comprising a plurality of memory slices, afirst array of values to one of a plurality of vector processors. Themethod can also include providing, by the memory fabric, a second arrayof values to a hardware accelerator comprising a programmable datapathpipeline, executing, by one of the plurality of vector processors, aninstruction that operates on the first array of values, configuring thedatapath pipeline in the hardware accelerator using configurationinformation, and performing, using the datapath pipeline in the hardwareaccelerator, a filtering operation on the second array of values inaccordance with the configuration information.

In some embodiments, the method can include receiving, at an outputbuffer, a scan-line of an image processed by the programmable datapathpipeline; and stalling, by a pipeline stall controller, an operation ofthe programmable datapath pipeline when the output buffer is full.

In some embodiments, the hardware accelerator comprises a plurality offunctional units, and the method includes chaining the plurality offunctional units in accordance with the configuration information toperform the filtering operation.

In some embodiments, the plurality of functional units comprises a firstfunctional unit and a second functional unit, and wherein chaining theplurality of functional units comprises an output of the firstfunctional unit to an input of the second functional unit.

In some embodiments, the method can include receiving depth informationthat is indicative of a depth of an object represented by a pixel in thescan-line of the image; and synchronizing the depth information to thescan-line of the image.

In some embodiments, the method can include monitoring, by a memorycontroller in the memory fabric, a status of an exclusive access requestrequesting an exclusive access to a shared resource by one of the vectorprocessors, and when the one of the vector processors receives anexclusive access to the shared resource, sending an acknowledgementmessage to the one of the vector processors, indicating that the one ofthe vector processors has the exclusive access to the shared resource.

In some embodiments, the memory fabric can include a plurality ofbuffers, wherein a first of the plurality of buffers is associated witha first of the vector processors, and the method further comprisessending, by a second of the vector processors, data to the first of thevector processor by storing the data in the first of the plurality ofbuffers.

In some embodiments, the method can include dynamically modifying acapacity of the first of the plurality of buffers based on an amount ofdata transferred to the first of the vector processors.

In some embodiments, the method can include dynamically associating twoor more of the plurality of buffers to the first of the vectorprocessors based on an amount of data transferred to the first of thevector processors.

In some embodiments, the method can include storing state information ofone of the vector processors in the memory fabric when the one of thevector processors enters a low-power mode.

In some embodiments, the state information is stored in a static randomaccess memory in the memory fabric.

In some embodiments, the method can include maintaining, at a directmemory access (DMA) controller, an operation list indicating an order inwhich DMA operations are to be performed.

In some embodiments, the method can include performing a subset of theDMA operations in the operation list based on an enable buffer, whereinthe enable buffer includes a plurality of bits, wherein one of theplurality of bits is associated with one of the DMA operations, and avalue of the one of the plurality of bits is indicative of whether theone of the DMA operations is to be performed by the DMA controller.

DESCRIPTION OF DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify like elements. The accompanying figures are schematic and arenot intended to be drawn to scale. For purposes of clarity, not everycomponent is labeled in every figure. Nor is every component of eachembodiment of the disclosed subject matter shown where illustration isnot necessary to allow those of ordinary skill in the art to understandthe disclosed subject matter.

FIG. 1 provides a high level illustration of a computing device inaccordance with some embodiments.

FIG. 2 illustrates a detailed illustration of a computing device inaccordance with some embodiments.

FIG. 3 illustrates a hardware accelerator in accordance with someembodiments.

FIG. 4 illustrates a hardware accelerator that can adapt a filteringoperation based on depth information in accordance with someembodiments.

FIG. 5 illustrates a hardware accelerator.

FIG. 6 illustrates a hardware accelerator based on generic functions inaccordance with some embodiments.

FIG. 7 illustrates a hardware accelerator that includes afirst-in-first-out (FIFO) buffer for communication between image signalprocessing (ISP) function modules in accordance with some embodiments.

FIG. 8 illustrates power supply gating of a power island in accordancewith some embodiments.

FIG. 9 illustrates a valid signal generator in accordance with someembodiments.

FIG. 10 illustrates an event signal monitoring mechanism in accordancewith some embodiments.

FIG. 11 shows a software defined interface in accordance with someembodiments.

FIG. 12 shows a detailed implementation of a software defined interfacein accordance with some embodiments.

FIG. 13 illustrates an event processor in accordance with someembodiments.

FIG. 14 illustrates an event filter in an event processor in accordancewith some embodiments.

FIG. 15 shows a bypass mode of a peripheral device in accordance withsome embodiments.

FIG. 16 shows a programmable Mobile Industry Processor Interface (MIPI)interface in accordance with some embodiments.

FIG. 17 illustrates an application of a polarity reversal mechanism foran input/output interface in accordance with some embodiments.

FIG. 18 illustrates a memory fabric having a hardware-based mutualexclusion (mutex) controller in accordance with some embodiments.

FIG. 19 illustrates a dynamic assignment of buffers in accordance withsome embodiments.

FIG. 20 illustrates a power management mechanism that provides differentvoltages to logic circuits memory devices in accordance with someembodiments.

FIG. 21 illustrates a direct memory access (DMA) engine that implementsa buffer-based DMA data structure enable mechanism in accordance withsome embodiments.

FIG. 22 illustrates an electronic device that includes the computingdevice in accordance with some embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forthregarding the systems and methods of the disclosed subject matter andthe environment in which such systems and methods may operate, etc., inorder to provide a thorough understanding of the disclosed subjectmatter. It will be apparent to one skilled in the art, however, that thedisclosed subject matter may be practiced without such specific details,and that certain features, which are well known in the art, are notdescribed in detail in order to avoid complication of the disclosedsubject matter. In addition, it will be understood that the examplesprovided below are exemplary, and that it is contemplated that there areother systems and methods that are within the scope of the disclosedsubject matter.

Computational imaging can transform the ways in which machines captureand interact with the physical world. For example, via computationalimaging, machines can capture images that were extremely difficult tocapture using traditional imaging techniques. As another example, viacomputational imaging, machines can understand their surroundings andreact in accordance with their surroundings.

One of the challenges in bringing computational imaging to a mass marketis that computational imaging is inherently computationally expensive.Computational imaging often uses a large number of images at a highresolution and/or a large number of videos with a high frame rate.Therefore, computational imaging often needs the support of powerfulcomputing platforms. Furthermore, because computational imaging is oftenused in mobile settings, for example, using a smart phone or a tabletcomputer, computational imaging often needs the support of powerfulcomputing platforms that can operate at a low power budget.

The present application discloses a computing device that can provide alow-power, highly capable computing platform for computational imaging.FIG. 1 provides a high level illustration of a computing device inaccordance with some embodiments. The computing device 100 can includeone or more processing units, for example one or more vector processors102 and one or more hardware accelerators 104, an intelligent memoryfabric 106, a peripheral device 108, and a power management module 110.

The one or more vector processors 102 includes a central processing unit(CPU) that implements an instruction set containing instructions thatoperate on an array of data called vectors. More particularly, the oneor more vector processors 102 can be configured to perform genericarithmetic operations on a large volume of data simultaneously. In someembodiments, the one or more vector processors 102 can include a singleinstruction multiple data, very long instruction word (SIMD-VLIW)processor. In some embodiments, the one or more vector processors 102can be designed to execute instructions associated with computer visionand imaging applications.

The one or more hardware accelerators 104 includes computer hardwarethat performs some functions faster than is possible in software runningon a more general-purpose CPU. Examples of a hardware accelerator innon-vision applications include a blitting acceleration module ingraphics processing units (GPUs) that is configured to combine severalbitmaps into one using a raster operator.

In some embodiments, the one or more hardware accelerators 104 canprovide a configurable infrastructure that is tailored to imageprocessing and computer vision applications. The hardware accelerators104 can be considered to include generic wrapper hardware foraccelerating image processing and computer vision operations surroundingan application-specific computational core. For example, a hardwareaccelerator 104 can include a dedicated filtering module for performingimage filtering operations. The filtering module can be configured tooperate a customized filter kernel across an image in an efficientmanner. In some embodiments, the hardware accelerator 104 can output onefully computed output pixel per clock cycle.

The intelligent memory fabric 106 can be configured to provide a lowpower memory system with small latency. Because images and videosinclude a large amount of data, providing a high-speed interface betweenmemory and processing units is important. In some embodiments, theintelligent memory fabric 106 can include, for example, 64 blocks ofmemory, each of which can include a 64-bit interface. In suchembodiments, the memory fabric 106 operating at 600 MHz, for example, iscapable of transferring data at 307.2 GB/sec. In other embodiments, theintelligent memory fabric 106 can include any other number of blocks ofmemory, each of which can include any number of interfaces implementingone or more interface protocols.

The peripheral device 108 can be configured to provide a communicationchannel for sending and receiving data bits to and from externaldevices, such as an image sensor and an accelerometer. The peripheraldevice 108 can provide a communication mechanism for the vectorprocessors 102, the hardware accelerators 104, and the memory fabric 106to communicate with external devices.

The power management module 110 can be configured to control activitiesof designated blocks within the computing device 100. More particularly,the power management module 110 can be configured to control the powersupply voltage of designated blocks, also referred to as power islands,within the computing device 100. For example, when the power managementmodule 110 enables a power supply of a power island, the computingdevice 100 can be triggered to provide an appropriate power supplyvoltage to the power island. In some embodiments, each power island caninclude an independent power domain. Therefore, the power supply ofpower islands can be controlled independently. In some embodiments, thepower management module 110 can also be configured to control activitiesof power islands externally attached to the computing device 100 via oneor more of input/output pins in the computing device 100.

FIG. 2 illustrates a detailed illustration of a computing device inaccordance with some embodiments. The computing device 100 can include aplurality of vector processors 102. In this illustration, the computingdevice 100 includes 12 vector processors 102. The vector processors 102can communicate with one another via the inter-processor interconnect(IPI) 202. The vector processors 102 can also communicate with othercomponents in the computing device 100, including the memory fabric 106and/or hardware accelerators 104, via the IPI 202 and the AcceleratorMemory Controller (AMC) crossbar 204 or a memory-mapped processor bus208.

In some embodiments, the one or more vector processors 102 can bedesigned to execute a proprietary instruction set. The proprietaryinstruction set can include a proprietary instruction. The proprietaryinstruction can be a variable length binary string that includes aninstruction header and one or more unit instructions. The instructionheader can include information on the instruction length and the activeunits for the associated proprietary instruction; the unit instructioncan be a variable length binary string that includes a number of fieldsthat are either fixed or variable. The fields in the unit instructioncan include an opcode that identifies the instruction and an operandthat specifies the value use in the unit instruction execution.

Details of the vector processors 102 are provided in U.S. PatentApplication No. TBD, entitled “VECTOR PROCESSOR,” identified by anAttorney Docket No. 2209599.127US1, filed on an even date herewith,which is herein incorporated by reference in its entirety.

The computing device 100 can include hardware accelerators 104. Thehardware accelerators 104 can include a variety of accelerator modulesthat are configured to perform predefined processing functions. In someembodiments, a predefined processing function can include a filteringoperation. For example, the hardware accelerators 104 can include a rawimage processing module, a lens shading correction (LSC) module, a bayerpattern demosaicing module, a sharpen filter module, a polyphase scalermodule, a Harris corner detection module, a color combination module, aluma channel denoise module, a chroma channel denoise module, a medianfilter module, a look-up table, a convolution module, an edge detectionmodule, and/or any other suitable module or combination of modules. Thehardware accelerators 104 can be configured to retrieve and store datain memory devices residing in the memory fabric 106.

The memory fabric 106 can include a central memory system thatcoordinates memory operations within the computing device 100. Thememory fabric 106 can be designed to reduce unnecessary data transferbetween processing units, such as vector processors 102 and hardwareaccelerators 104. The memory fabric 106 is constructed to allow aplurality of processing units to access, in parallel, data and programcode memory without stalling. Additionally, the memory fabric 106 canmake provision for a host processor to access the memory system in thememory fabric 106 via a parallel bus such as the Advanced eXtensibleInterface (AXI) or any other suitable bus 208.

In some embodiments, a processing unit can read/write up to 128-bits percycle through its load-store unit (LSU) ports and read up to 128 bitprogram code per cycle through its instruction port. In addition to IPI202 and AMC 204 interfaces for processors 102 and hardware accelerators104, respectively, the memory fabric 106 can provide simultaneousread/write access to a memory system through the AdvancedMicrocontroller Bus Architecture (AMBA) High-performance Bus (AHB) andAXI bus interfaces. The AHB and AXI are standard parallel interfacebuses which allow processing units, a memory system, and a peripheraldevice to be connected using a shared bus infrastructure. Any othersuitable buses can be used. In some embodiments, the memory fabric 106can be configured to handle a peak of 18×128-bit memory accesses perclock cycle. In other embodiments, the memory fabric 106 can be designedto handle any number of memory accesses per clock cycle using ahigh-speed interface with a large number of bits.

A memory system in the memory fabric 106 can include a plurality ofmemory slices, each memory slice being associated with one of the vectorprocessors 102 and giving preferential access to that processor overother vector processors 102. Each memory slice can include a pluralityof Random Access Memory (RAM) tiles, where each RAM tile can include aread port and a write port. In some cases, each memory slice may beprovided with a memory slice controller for providing access to arelated memory slice.

The processors and the RAM tiles can be coupled to one another via abus, also referred to as an IPI 202. In some cases, the IPI 202 cancouple any of the vector processors 202 with any of the memory slices inthe memory fabric 106. Suitably, each RAM tile can include a tilecontrol logic block for granting access to the tile. The tile controllogic block is sometimes referred to as tile control logic or anarbitration block.

In some embodiments, each memory slice can include a plurality of RAMtiles or physical RAM blocks. For instance, a memory slice having thesize of 128 kB can include four 32 kB single-ported RAM tiles (e.g.,physical RAM elements) organized as 4 k×32-bit words. As anotherinstance, a memory slice having a size of 256 kB can include eight 32 kBsingle-ported RAM tiles (e.g., physical RAM elements) organized as 8k×32-bit words. In some embodiments, the memory slice can have acapacity as low as 16 kB and as high as 16 MB. In other embodiments, thememory slice can be configured to have as much capacity as needed toaccommodate a variety of applications handled by the computing device.

In some embodiments, a RAM tile can include a single portedcomplementary metal-oxide-semiconductor (CMOS) RAM. The advantage of asingle ported CMOS RAM is that it is generally available in mostsemiconductor processes. In other embodiments, a RAM tile can include amulti-ported CMOS RAM. In some embodiments, each RAM tile can be 16-bitwide, 32-bit wide, 64-bit wide, 128-bit wide, or can be as wide asneeded by the particular application of the computing device.

The use of single-ported memory devices can increase the power and areaefficiency of the memory subsystem but can limit the bandwidth of thememory system. In some embodiments, the memory fabric 106 can bedesigned to allow these memory devices to behave as a virtualmulti-ported memory subsystem capable of servicing multiple simultaneousread and write requests from multiple sources (processors and hardwareblocks). This can be achieved by using multiple physical RAM instancesand providing arbitrated access to them to service multiple sources.

In some embodiments, each RAM tile can be associated with tile controllogic. The tile control logic is configured to receive requests fromvector processors 102 or hardware accelerators 104 and provide access toindividual read and write-ports of the associated RAM tile. For example,when a vector processor 102 is ready to access data in a RAM tile,before the vector processor 102 sends the memory data request to the RAMtile directly, the vector processor 102 can send a memory access requestto the tile control logic associated with the RAM tile. The memoryaccess request can include a memory address of data requested by theprocessing element. Subsequently, the tile control logic can analyze thememory access request and determine whether the vector processor 102 canaccess the requested RAM tile. If the vector processor 102 can accessthe requested RAM tile, the tile control logic can send an access grantmessage to the vector processor 102, and subsequently, the vectorprocessor 102 can send a memory data request to the RAM tile.

In some embodiments, the tile control logic can be configured todetermine and enforce an order in which many processing units (e.g.,vector processors and hardware accelerators) access the same RAM tile.For example, the tile control logic can include a clash detector, whichis configured to detect an instance at which two or more processingunits attempt to access a RAM tile simultaneously. The clash detectorcan be configured to report to a runtime scheduler that an access clashhas occurred and that the access clash should be resolved.

The memory fabric 106 can also include a memory bus for transferringdata bits from memory to vector processors 102 or hardware accelerators104, or from vector processors 102 or hardware accelerators 104 tomemory. The memory fabric 106 can also include a direct memory access(DMA) controller that coordinates the data transfer amongst vectorprocessors 102, hardware accelerators 104, and memory.

In some embodiments, the hardware accelerators 104 can be coupled to thememory fabric 106 via a separate bus. The separate bus can include anaccelerator memory controller (AMC) 204, which is configured to receiverequests from at least one hardware accelerator and to grant, to thehardware accelerator, an access to a memory slice through the relatedmemory slice controller. It will thus be appreciated that the memoryaccess path employed by the hardware accelerators 104 can be differentto the path employed by the vector processors 102. In effect, the AMC204 can perform address filtering, arbitration and multiplexing. In someembodiments, the hardware accelerators 104 can include an internalbuffer (e.g., a FIFO memory) to account for delays in accessing thememory fabric 106.

In some embodiments, the AMC 204 may be coupled to one or moreperipheral devices 108, including, for example, a plurality of MobileIndustry Processor Interface (MIPI) camera interfaces. The AMC 204 canalso be connected to AXI and APB interfaces to allow two system RISCprocessors to access memory slices in the memory fabric 106 via the AMC204.

In some embodiments, the AMC 204 can include a pair of 64 bit ports intoeach memory slice of the memory fabric 106. The AMC 204 can beconfigured to route requests from a hardware accelerator 104 to anappropriate memory slice by partial address decode.

In some embodiments, the AMC 204 can be coupled to a wide variety ofprocessing units to provide access to memory slices in the memory fabric106. For example, the AMC 204 may be coupled to any type of hardwareaccelerators or 3rd party elements to provide access to memory slices inthe memory fabric 106. The AMC 204 may also be configured to provideaccess to a wider memory space of a computing system, including memorydevices that lie outside of the computing device 100.

In some embodiments, the AMC 204 can arbitrate simultaneous memoryaccess requests to the same memory slice in a round-robin manner. Forexample, a processing unit, such as a hardware accelerator 104, cansend, to the AMC 204, a memory access request, which includes a memoryaddress. When the AMC 204 receives the memory access request, the AMC204 determines whether the memory address in the memory access requestis associated with a memory slice in the memory fabric 106. If thememory address in the memory access request is not associated with amemory slice in the memory fabric 106, then the AMC 204 can forward thememory request to the AMC's AXI master. If the memory address in thememory access request is associated with a memory slice in the memoryfabric 106, the AMC 204 can arbitrate the memory access request toprovide access to the desired memory location.

The peripheral device 108 can be configured to provide a communicationchannel for sending and receiving data bits to and from externaldevices, such as multiple heterogeneous image sensors and anaccelerometer. The peripheral device 108 can provide a communicationmechanism for the vector processors 102, the hardware accelerators 104,and the memory fabric 106 to communicate with external devices.

Traditionally, the functionality of a peripheral device has been fixedand hard-coded. For example, mobile industry processor interface (MIPI)peripherals were only able to interface with an external device thatalso implements lower-rate digital interfaces such as the SPI, I2C, I2S,or any other suitable standards.

However, in some embodiments of the present disclosure, thefunctionality of the peripheral device 108 may be defined usingsoftware. More particularly, the peripheral device 108 can include anemulation module that is capable of emulating the functionality ofstandardized interface protocols, such as SPI, I2C, I2S, or any othersuitable protocol.

The power management module 110 is configured to control activities ofblocks within the computing device 100. More particularly, the powermanagement module 110 is configured to control the power supply voltageof designated blocks, also referred to as power islands. For example,when the power management module 110 enables a power supply of a powerisland, the computing device 100 is configured to provide an appropriatepower supply voltage to the power island. The power management module110 can be configured to enable a power supply of a power island byapplying an enable signal in a register or on a signal line on a bus. Insome embodiments, the power management module 110 can also be configuredto control activities of external device via one or more of input/outputpins in the computing device 100.

In some embodiments, a power island can be always powered-on (e.g., thepower supply voltage is always provided to the power island). Such apower island can be referred to as an always-on power island. In someembodiments, the always-on power-island can be used to monitor signalsfrom, for example, General-Purpose-Input-Output (GPIO) pins, externalinterfaces, and/or internal functional blocks such as a low frequencytimer or power-on reset. This way, the computing device 100 can respondto an event or a sequence of events and adaptively power-up only thepower-islands that are needed to respond to the event or the sequence ofevents.

FIG. 3 illustrates a hardware accelerator in accordance with someembodiments. The hardware accelerator 104 can include a collection ofhardware image processing filters. The hardware accelerator 104 canenable some of the computationally intensive functionalities to beoffloaded from the vector processors 102. The accelerator 104 can becoupled to the AMC 204 to access memory slices in the memory fabric 106at a high bandwidth.

In some embodiments, the hardware accelerator 104 can be coupled to thememory fabric 106 via the AMC 204. In some embodiments, the hardwareaccelerator 104 can include one or more filter modules (e.g., 20 filtermodules), including a MIPI receiver filter and a MIPI transmitterfilter. In some embodiments, a filter module may include one read-onlyAMC interface (a read client interface) and one write-only AMC interface(a write client interface). In other embodiments, a filter module canalso have a plurality of read-only AMC interfaces. For example, a filtermodule may have a plurality of read-only AMC interfaces for a parallelaccess to multiple input buffers, multiple planes (from the samebuffer). The plurality of read-only AMC interface can be used to providean extra memory read bandwidth to sustain the filter module's processingthroughput. The descriptions of a hardware accelerator 104 can beequally applicable to each filter module since a hardware accelerator104 may only have a single filter module. Likewise, the descriptions ofa filter module can be equally applicable to a hardware acceleratorsince the filter module may be the only filter module in the hardwareaccelerator.

In some embodiments, the AMC 204 has one or more bi-directional (e.g.,read/write) ports into each memory slice in the memory fabric 106. Theports can accommodate a large number of bits. For example, the ports canaccommodate a 64-bit communication. In some embodiments, the AMC 204 canalso include an AXI master, which provides a direct connectivity toexternal DRAM devices.

In some embodiments, a filter module can be designed primarily toprocess buffers in the memory fabric 106. For example, with theexception of a MIPI receiver module and a MIPI transmitter filtermodule, a filter module can input and output data only via its AMCclients. The configuration of filter modules, including their bufferbase addresses, can be achieved via several APB slave interfaces.

In some embodiments, the hardware accelerator 104 can receive image datavia a MIPI receiver filter module and a MIPI transmitter filter module.The MIPI receiver filter module and the MIPI transmitter filter modulecan allow other filter modules in the hardware accelerator 104 toestablish a direct connection to a MIPI receiver controller and a MIPItransceiver controller. The MIPI receiver filter module and the MIPItransmitter filter module can connect to the MIPI controllers viaparallel interfaces and can be used to stream data into/out of thememory fabric 106 directly from/to the MIPI Controller.

In some embodiments, the hardware accelerator 106 can operate onscan-lines of image data buffered in the memory fabric 106, accessed viathe AMC 204. The AMC 204 can route transactions from its clientinterfaces to the target memory slice (or the AXI master) and arbitratebetween simultaneous transactions from different clients at each memoryslice. In some embodiments, multiple filter modules in the hardwareaccelerator 106 may be connected together in a streaming fashion bycoupling an output buffer of one or more filter modules (also referredto as producers/parents) to input buffers of other filter modules (alsoreferred to as consumers/children).

In some embodiments, a filter module in a hardware accelerator 104 canoperate a 2-dimensional kernel on pixels centered at the current pixel.All the pixels in the kernel can contribute in processing pixelscentered at the current pixel.

In some embodiments, a filter module in a hardware accelerator 104 canprocess an image line-by-line. For example, a filter module can scan animage from the top to bottom to generate a scan-line of an image, andprocess the scan-lines, for instance, moving from left to right. Inother examples, a filter module can generate scan-lines of an image byscanning an image in any orientation/ordering suitable for the filterprocessing.

In some embodiments, a filter module can process a scan-line of an imageby reading data to form a kernel for a first pixel on the scan-line. Thefilter module can process the scan-line by sliding the kernel in asliding-window manner. Once the processing is complete, the filtermodule can write the output pixels into an output buffer or a memorylocation.

In some embodiments, kernels for filtering are typically square andoften have an odd number of pixels along each side, e.g. 3×3, 5×5, or7×7. If a filter module uses a K×K pixel kernel, then K scan-lines ofimage data can be read from an input buffer for each line of image dataprocessed and written to the its output buffer.

In some embodiments, the hardware accelerator 104 can use a circularinput buffer. Suppose that a target filter module is configured toreceive, as input, an output scan-line of another filter module (alsoreferred to as a parent filter module). Suppose also that the targetfilter module uses a K×K pixel kernel. Then the input buffer for thetarget filter module can be designed to maintain at least (K+1)scan-lines of image data: K scan-lines for the filter module and one (ormore) scan-line for simultaneously receiving an output scan-line of theparent filter module. In this example, because the input buffer iscircular, after receiving (K+1) scan-lines from the parent filtermodule, the (K+2)th scan-line can be written over the location of thefirst line. In most cases, the parent filter module can be ahead of thetarget filter module in terms of its current line number within theinput image. After the initial configuration, the filter modules' readand write AMC clients can take care of circular buffer address wrappingwhen accessing the filter modules' input and output buffers.

In some embodiments, buffers in the hardware accelerator 104 can bealigned by a predetermined number of bytes. For example, buffers in thehardware accelerator 104 can be aligned on 8-byte boundaries. To easethe transaction routing, the read and write clients and the AMC can beconfigured to provide only aligned buffer accesses. When an image widthis not a multiple of a predetermined number of bytes, then the hardwareaccelerator 104 can be configured to write null bytes to output buffersbetween the (unaligned) end of each scan-line and the next byteboundary.

FIG. 3 illustrates an implementation of a hardware accelerator foroperating a filter kernel, stored in a filter kernel register 302, on aninput data stream (e.g., scan-lines of one or more images). The inputdata streams can correspond to pixels in one or more images. Thehardware accelerator 104 can include a datapath pipeline 304, a pipelinestall controller 306, a line buffer read client 308, a line startcontrol input 310, and a line buffer write client 310. In someembodiments, a hardware accelerator 104 can include at least one AMCread client interface 314 and/or at least one AMC write client interface316 to access a memory slice in the memory fabric 106. The number ofread/write client interfaces on the AMC 204 is suitably configurable.

In some embodiments, the filter kernel register 302 can be programmed tomodify the kernel to be operated on the input data stream. The filterkernel register 302 can be configured to accommodate a variety of kernelsizes. For example, the filter kernel register 302 can be configured toaccommodate a 3×3 kernel, a 5×5 kernel, a 7×7 kernel, a 9×9 kernel, orany other kernel sizes represented as m×n. In some cases, m can be thesame as n; in other cases, m can be different from n. In someembodiments, the filter kernel register 302 can be configured toaccommodate kernels of various dimensions. For example, the filterkernel register 302 can be configured to accommodate a one-dimensionalfilter, a two-dimensional filter, a three-dimensional filter, or anyinteger-dimensional filters.

In some embodiments, the line buffer read client 308 is configured toreceive a scan-line of an image (e.g., a row or a column of an image onan image grid) and to provide the scan-line to the datapath pipeline304. The line buffer read client 308 can receive the scan-line of animage via an AMC read interface 314. Once the datapath pipeline 304receives a kernel and a scan-line of an image, the datapath pipeline 304can perform the filtering operation. Once the datapath pipeline 304completes the filtering operation, the datapath pipeline 304 can storethe resulting line in the line-buffer write client 312. The line bufferwrite client 312 can, optionally, store the resulting line in a memoryslice via an AMC write interface 316. The pipeline stall controller 306can stall certain parts of the pipeline to ensure that the line-bufferwrite client 312 does not overflow.

In some embodiments, the line start controller 310 can control a timeinstance at which the datapath pipeline 304 starts processing thereceived scan-line of an image. The line start controller 310 can alsobe configured to selectively enable one or more portions of the datapathpipeline 304 to perform customized operations. In some cases, the linestart controller 310 can also control coefficients to be used during thefiltering operation by the datapath pipeline 304.

In some embodiments, the datapath pipeline 304 and the line startcontroller 310 can be programmable. The datapath pipeline 304 and theline start controller 310 can be programmed so that different types offiltering operations can be performed by the hardware accelerator 104.For example, the datapath pipeline 304 and the line start controller 310can be programmed with filter operation parameters, such as coefficientsets and/or thresholds, so that customized filtering operation can becarried out by the hardware accelerator 104. The filter operationparameters can also include a filter kernel size, coefficients, scalingratios, gains, thresholds, look-up tables, or any other suitableparameters. Therefore, the hardware accelerator 104 can be considered asa generic wrapper for accommodating various image filtering operations.

In some embodiments, the datapath pipeline 304 can be configured toprocess numbers represented in one or more number formats. For example,the datapath pipeline 304 can be designed to operate on floating pointnumbers, e.g., fp 16 (IEEE754-like 16-bit floating-point format),integer numbers, fixed-point numbers, or any other number formatssuitable for image processing.

The hardware accelerator 104 can be configured to control how thedatapath pipeline 304 consumes scan-lines from an input data buffer 308and how the datapath pipeline 304 stores processed scan-lines to anoutput data buffer 312. The hardware accelerator 104 can be configuredto implement one of two control modes: the buffer fill control (BFC)mode and the synchronous mode.

In some embodiments, under BFC mode, the hardware accelerator 104 can beconfigured to maintain internal counts of fill levels (e.g., the numberof scan-lines stored in the input buffer). The hardware accelerator 104can be configured to process a scan-line from its input bufferautonomously when (1) the hardware accelerator is enabled, (2) its inputbuffer has sufficient number of scan-lines, and (3) there is space inits output buffer to store a processed scan-line. In some cases, thebuffer fill level needed to run the datapath pipeline 304 can depend onthe height of a kernel. For example, when a kernel is 3×3, then thehardware accelerator 104 can require at least three scan-lines tooperate a filter.

In some embodiments, under a synchronous control mode, a filter modulein a hardware accelerator can be configured to run when a start bit forthe filter module is turned on. The start bit can be turned on using,for example, a software module. Under synchronous control, the softwaremodule can be configured to determine that the input buffer for thefilter module has a sufficient number of scan-lines and that the outputbuffer for the filter module has sufficient space to store processedscan-lines from the filter module. Once these conditions are satisfied,the software module can turn on the start bit of the filter module.

Under both modes, once a filter module processes a scan-line, the filtermodule can update its current line index within its buffer and withinthe input image. In some embodiments, when the output image does nothave the same size as the input image, the filter module can update itscurrent line index in the output image as well. The values of the lineindices (and buffer fill levels for buffer fill control) can representthe internal state of a filter module. This internal state can beaccessed by a software module and may be saved, updated, and restoredsuch that the context of the filter module may be switched before thefilter module is run in the next cycle.

In some embodiments, buffers in a hardware accelerator 104 can beconfigured to maintain a plurality of data planes. For example, buffersin a hardware accelerator 104 can be configured to maintain thered-channel, the green-channel, and the blue-channel of an image inseparate planes. In some examples, the buffers in a hardware accelerator104 can be configured to support up to sixteen planes. The scan-lines ofan image data in each plane can be stored contiguously and planes can bedefined by their number and by a plane stride.

In some embodiments, a filter module in a hardware accelerator 104 canbe configured to process a scan-line from each data plane sequentially,one at a time. For sequential processing, from the control point ofview, scan-lines from all planes may be considered to have the same timestamp. In other embodiments, a filter module in a hardware accelerator104 can be configured to process multiple data planes in parallel.

In some embodiments, prior to processing an image/video stream, or ifcontext is switched, a filter module can be appropriately configured andenabled. Each filter module can include a set of software programmableregisters defining its input buffer(s) and output buffer configuration.

In some embodiments, a buffer in a filter module can be programmed usingone or more of following parameters:

-   -   base: Base address. This parameter can specify the base address        of the buffer. Addresses can be aligned on byte boundaries        (e.g., the width of the AMC client data bus).    -   nl: Number of scan-lines. In circular buffer mode, this        parameter can specify the size of a circular buffer in        scan-lines. The maximum number of scan-lines for a circular        buffer can be 1023, but other upper bounds are also possible. If        a buffer is configured with nl=0, it indicates that the buffer        is in a non-circular mode. Therefore, nl=0 puts the read/write        client(s) accessing the buffer into non-circular or no-wrap mode        in which the number of scan-lines in the buffer corresponds to        the height of the image and no circular buffer pointer wrapping        occurs.    -   ls: Line stride. The line stride can be a multiple of a fixed        number of bytes, for example, 8 bytes. The maximum line stride        can be predetermined. For example, the maximum line stride can        be (32 MB-8) bytes. The line stride and number of lines can be        used by read/write clients to perform circular buffer pointer        arithmetic. The line stride can be greater than or equal to the        image width.    -   np: Number of planes. This parameter indicates a number of        planes represented by a buffer. When np=0, it indicates that the        buffer represents non-planar data (e.g., a single plane data).        The amount of line buffer storage in a buffer can be multiplied        by the number of planes.    -   ps: Plane stride. The plane stride can be a multiple of a fixed        number of bytes, for example, 8 bytes. The maximum plane stride        can be predetermined. For example, the maximum plane stride can        be (32 MB-8) bytes. Normally, the plane stride can be greater        than or equal to nl multiplied by ls. However, other plane        stride can be possible.    -   format: Buffer data format. This parameter can specify the size        of the pixel data in bytes. For example, for an FP16 buffer, the        format can be set to 2, indicating 2 bytes per pixel.

In some embodiments, an output buffer in a filter module can beprogrammed using one or more of following parameters:

-   -   offset: The offset can specify the offset from the base address        (and the start of each line) to the first pixel. This parameter        may be used to work-around the limitation of buffers being        aligned on a byte boundary. Using the offset, a space may be        reserved on the left of scan-lines, for example for horizontal        pixel padding by an output buffer's consumer. The default offset        is zero. If a non-zero offset is specified, then the null bytes        can be written to each output scan-line before the first output        pixel.

In some embodiments, a filter module may support a variety of datatypes. The most common data types supported by a filter module arelisted below:

-   -   U8—unsigned 8 bit integer data    -   U8F—unsigned 8 bit fractional data the range [0, 1.0]    -   U16—unsigned 16 bit integer data    -   U32—unsigned 32 bit integer data    -   FP16—half-precision (16 bit) floating point    -   FP32—full-precision (32 bit) floating point

In some embodiments, the datapath pipeline of a filter module can beoptimized for its operation: half-precision floating point (FP16)arithmetic can used for operations involving a high dynamic range;optimized fixed-point arithmetic can be used where maintaining highprecision is more important.

In some embodiments, a filter module implemented using a FP16 arithmeticmay not be restricted to reading/writing only to FP16 buffers. U8Fbuffers may also be accessed with conversion to/from FP16 taking placeautomatically within the filter modules.

In some embodiments, where a filter module is implemented using FP16arithmetic, the buffers may be either FP16 or U8F. When a buffer isFP16, the buffer configuration format can be set to 2. If a buffer isU8F, the buffer configuration format can be set to 1. For filter moduleswith FP16 datapath pipeline, if the input buffer format is “1,” the readclient can convert the U8F input data to FP16 automatically beforeprocessing. If the output buffer format is “1,” the write client canconvert FP16 from the datapath pipeline to U8F before storage.

In some embodiments, U8F is converted to normalized FP16, in the range[0, 1.0], by multiplying by 1.0/255. Normalized FP16 can be converted toU8F by multiplying by 255 and rounding, effectively quantizing thefloating-point values into 8 bits. In some embodiments, the output datafrom filter modules with FP16 datapath pipeline may optionally beclamped into the normalized range [0, 1.0]. If conversion to U8F isenabled, then the clamp to the normalized range is implicitly enabledand is performed prior to the conversion to U8F described above. Filtermodules implemented using FP16 datapath pipelines are not limited toprocessing data in the normalized range [0, 1.0]; the full range of FP16can also be supported.

In some embodiments, a filter module is configured to track its verticalposition in an input image. A filter module can use this information toperform vertical padding at the top and bottom of the image by linereplication or reflection. A filter module that does not performvertical padding may create an output image that is smaller than aninput image, which may not be desirable in some cases.

In some embodiments, when a filter module is configured to performvertical padding, the minimum number of scan-lines M that can bemaintained by an input buffer can be:

M=(K>>1)+1,

where >> indicates a right bit-shift operator.At the top of the image, when the capacity of the input buffer (in termsof scan-lines) is less than M, there are not enough scan-lines in thebuffer to perform the filtering operation. When the capacity of theinput buffer (in terms of scan-lines) is greater than or equal to M,data may be processed if vertical padding is performed. Similarly, atthe bottom of the image, when processing the last (K>>1) lines, thefilter module can perform the replication of line N−1 (or reflection ofline N−1 and the lines above it).

In some embodiments, vertical padding can be performed when the kernelhas an even dimension. Vertical padding for a kernel with an evendimension can be virtually identical to vertical padding for a kernelwith an odd dimension, except that one less line should be padded at thebottom.

In some embodiments, a filter module can perform a horizontal padding.The horizontal padding of a pixel kernel can be performed as data isread from the input buffer and written to the pixel kernel registers.The filter module can be aware of its position on the current line andat the start and end of a line. Therefore, valid pixel kernel registerscan be replicated into those which do not hold valid data. As withvertical padding, whether horizontal padding is performed or not candepend on the specific functionality and requirements of a given filtermodule.

In some embodiments, in a circular buffer mode, a filter module can beconfigured to process one scan-line from its input buffer and write theprocessed scan-line to its output buffer. This set of operation can bereferred to as a filter run.

In some embodiments, for flexibility, two different control mechanismscan be provided by which filter runs may be controlled. In the firstmechanism, called buffer fill control mode, a filter module can trackthe fill levels of its circular buffers and determine, on its own,whether it can run. This approach is asynchronous in nature; the filtermodule can run, possibly repeatedly, as long as the required conditionsare met. Control bits in registers are provided to allow software toinform the filter modules when a scan-line has been added to an inputbuffer or removed from an output buffer. When a scan-line is added to aninput buffer, the fill level can be increased; when a scan-line isremoved from an output buffer, the fill level can be decreased. In thismode, a filter module, together with its input and output buffers, maybe viewed as a first-in-first-out (FIFO) with scan-lines occupying itsentries and the depth of the FIFO configured by the number of scan-linesprogrammed for the input and output buffers.

In some embodiments, another filter module may add a scan-line to theFIFO if the filter module's input buffer is not full. Software can checkthe fill level of an input buffer before allowing another filter moduleto add a scan-line to the input buffer. Subsequently, the software or afilter module can increase a fill level associated with the inputbuffer. On the output side, the software can check the fill level of theoutput buffer, or respond to an interrupt event signifying that a filtermodule has added a new scan-line to its output buffer, beforedecrementing the output buffer's fill level (e.g. after a line in thefilter's output buffer has been processed by another filter, likereading the FIFO).

The second mechanism, called a synchronous mode, depends on software toexplicitly schedule each filter run. Start bits for each filter modulecan be provided in registers to which software may write to start afilter run immediately. When started by this mechanism, a filter modulecan be executed exactly once.

In some embodiments, a filter module can be interrupted when it receivesan interrupt request. In some cases, a filter module can have aplurality of interrupt request sources which are mapped to externalinterrupt request lines and routed an interrupt controller. When afilter module flags an interrupt and that interrupt is enabled, then thecorresponding external interrupt request line can be flagged.

In some embodiments, the plurality of interrupt request sources caninclude:

-   -   Input buffer fill level decrement interrupt    -   Output buffer fill level increment interrupt    -   Frame done interrupt        The output buffer fill level increment interrupt may also be        deemed to indicate that a filter module has finished its filter        run when the filter module is configured to operate in        synchronous mode.

In some embodiments, the hardware accelerator 104 can adapt thefiltering operation based on depth information. For example, thehardware accelerator 104 can be configured to conditionally blur onlythe pixels associated with objects that are further than 30 yards awayor pixels that are beyond 5 yards could be blurred less than thosebeyond 10 yards, etc.

FIG. 4 illustrates a hardware accelerator that can adapt a filteringoperation based on depth information in accordance with someembodiments. The depth-aware hardware accelerator 402 includes, inaddition to modules in the hardware accelerator 104 in FIG. 3, a depthmap read client 404 and a depth map module 406. The depth map readclient 404 is configured to receive a depth map that indicates a depthof an object represented by a pixel in the corresponding image. Forexample, when the line buffer read client 308 receives a scan-line of animage, the depth map read client 404 can be configured to receive adepth map corresponding to the scan-line of the image.

Subsequently, the depth map read client 404 can provide the depth map tothe depth map module 406. When the resolution of the depth map is lowerthan the resolution of a scan-line of an image, the depth map module 406can be configured to up-sample the depth map to match the resolution ofthe depth map to the resolution of the scan-line. When the depth map isnot time-synchronized with a scan-line of an image, the depth map module406 can be configured to synchronize the depth map and the scan-line.The depth map module 406 can subsequently provide the processed depthmap to the line start controller 310 so that the line start controller310 can control the operation of the datapath pipeline 304. Moregenerally, an arithmetic function can be applied conditionally eitherbased on comparison of the depth at a pixel location to one or morethresholds using a comparator, or alternately, directly using a binarycontrol bit associated with each pixel which can be applied in place ofthe comparator output using a bypass multiplexer.

Traditionally, hardware accelerators for image processing operationsincluded a fixed set of hard-wired image signal processing (ISP)functions arranged in a predetermined order. FIG. 5 illustrates atraditional hardware accelerator. A traditional hardware accelerator 500would receive an image from a memory device 504, and process thereceived image using ISP functions 502A-502H in the order that ispredetermined at design time. In the example shown in FIG. 5, thehardware accelerator 500 uses 8 ISP functions in the illustrated orderto process the received image. This approach is rather inflexible andmay limit application areas in which the hardware accelerator 500 can beused. Image sensor technology is moving fast and it is difficult toenvision using a single fixed ISP pipeline for all current and futuresensors. Furthermore, when an ISP function operates a filter on multiplescan-lines of an image, the ISP function has to store, in a buffer,incoming scan-lines until sufficient number of scan-lines are present.These buffers are typically implemented using RAM devices sizedaccording to the resolution of the image, and the size of the buffer ispredetermined at the design time of the hardware accelerator 500.Therefore, the buffer for the ISP can effectively force a hard limit onthe image resolution that may be handled by the hardware accelerator500. Additionally, since the buffer is private to the ISP function, thebuffer cannot be used in other scenarios (for example by software) andcan consume a large amount of die area.

In some embodiments, the hardware accelerator 104 addresses theinflexibility of traditional hardware accelerators by chaining generic,common ISP functions. Frequently, the difference between hardwareaccelerators lie not so much in the functionality of ISP functionsimplemented by the hardware accelerators, but the order (and in somecases number of times) in which the ISP functions are invoked.Therefore, the hardware accelerator 104 can be configured to perform adesired function by chaining one or more generic, common functionmodules that are implemented efficiently.

For example, a convolution operation can be represented as amultiplication and a summation. Likewise, a finite impulse response(FIR) filtering operation can also be represented as a multiplicationand a summation, although the order in which the FIR filtering operationperforms the multiplication and summation may be different from that ofthe convolution operation. Despite the difference between theconvolution operation and the FIR filtering operation, themultiplication operation and the summation operation are the commonfunctions for the convolution operation and the FIR filtering operation.Therefore, the hardware accelerator 104 can be designed to perform theconvolution operation and the finite impulse response filteringoperation using the same multiplication module and the same summationmodule.

In some embodiments, the order in which the generic, common functionsare invoked can be determined using software. For example, software canprogram the hardware accelerator to invoke the multiplication module andthe summation module to perform either the convolution operation or theFIR filtering operation by chaining the multiplication module and thesummation module in a different order.

FIG. 6 illustrates a hardware accelerator based on generic functions inaccordance with some embodiments. The hardware accelerator 102 caninclude a plurality of generic ISP function modules 602A-602H, a datareceiver module 604 for receiving one or more scan-lines of an image forprocessing, and a data output module 606 for outputting one or morescan-lines that have been processed by one or more generic ISP functionmodules 602A-602H. In some embodiments, the one or more generic ISPfunction modules 602A-602H can include a configuration register and acontrol register. The values for these registers can be controlled usingsoftware. In some embodiments, the plurality of generic ISP functionmodules 602A-602H can be a part of the datapath pipeline 304.

In some embodiments, one or more of the generic ISP function modules602A-602H can include a self-contained hardware filter that alsoincludes direct memory access (DMA) capabilities. The one or more of thegeneric ISP function modules 602A-602H can use the DMA capabilities toload and/or store data from and/or to a memory slice in the memoryfabric 106. The DMA capabilities can be controlled using software.

In some embodiments, the data receiver module 604 can include a DMAmodule for retrieving one or more scan-lines of an image. In otherembodiments, the data receiver module 604 can include a sensor interfacemodule such as a MIPI module. In some embodiments, the data outputmodule 606 can include a DMA module for storing one or more processedscan-lines of an image. In other embodiments, the data output module 606can include a display device.

In some embodiments, the hardware accelerator 102 can be coupled to amemory fabric 106 that includes an ISP table. The ISP table can includeone or more buffers 608. Each buffer can include a pointer to one of thegeneric ISP function modules 602A-602H. Since the memory fabric 106 caninclude a multi-ported common (or uniform) memory, multiple devices canaccess the one or more buffers 608 in the ISP table to identifyavailable generic ISP function modules.

In some embodiments, software ISP functions 612A-612C, running on aprocessor 610, can be designed to execute one or more generic ISPfunction modules 602A-602H in the hardware accelerator 102. For example,a software ISP function 612A can determine (1) a list of generic ISPfunction modules 602A-602H to be executed to perform a desired functionand (2) an order in which the list of generic ISP function modules602A-602H should be executed. Then, the software ISP function 612A canuse one or more buffers 608 corresponding to the list of generic ISPfunction modules 602A-602H to chain the generic ISP function modules,thereby performing the desired function. In essence, the functionalityof the hardware accelerator can be determined by software in its look-upof the buffers 608 in the ISP table.

In some embodiments, an input interface of ISP function modules may bedirectly coupled to an output interface of other ISP function modules bymeans of a small memory mapped first-in-first-out (FIFO) buffer. FIG. 7illustrates a hardware accelerator that includes a FIFO buffer forcommunication between ISP function modules in accordance with someembodiments. The ISP function modules 602 can be coupled to a memory businterface 702, which is in turn coupled to a FIFO buffer 704 and amemory fabric 106.

When a first ISP function module 602A completes its operation on ascan-line of an image, the first ISP function module 602A can store theprocessed scan-line in a FIFO buffer 704. As the first ISP functionmodule 602A continues to process additional scan-lines, the first ISPfunction module 602A can continue to store the processed scan-lines inthe FIFO buffer 704 until the FIFO buffer 704 is full. When the FIFObuffer 704 is full, the first ISP function module 602A can be stalleduntil the FIFO buffer 704 is no longer full. In the meanwhile, a secondISP function module 602B can retrieve processed scan-lines from the FIFObuffer 704 for further processing, until the FIFO buffer 704 is empty.In effect, the first ISP function module 602A can be considered theproducer of data; the second ISP function module 602B can be consideredthe consumer of data; and the FIFO buffer 704 can be considered anarbitrator. Since the second ISP function module 602B can retrieveprocessed scan-lines from the FIFO buffer 704, which has a lower latencycompared to a memory slice in the memory fabric 106, the FIFO buffer 704can reduce the latency of a chain of ISP function modules 602.

In some embodiments, the computing device 100 can include a plurality ofpower islands. Each power island can be associated with a dedicatedpower domain. Therefore, the power supply voltage of each power islandcan be controlled independently. For example, the computing device 100can determine which power islands are needed to perform a certainoperation, and turn on the power supply voltage of only those powerislands that are needed. This way, the computing device 100 can reducethe leakage power consumption.

In some embodiments, when the computing device 100 determines that apower island is currently in a low-power mode (e.g., no power supplyvoltage is provided), and that the power island is needed for aparticular operation, the computing device 100 can invoke a power-upsequence for the power island and provide a power supply voltage to thepower island.

In some embodiments, each of the vector processors 102 can be associatedwith a unique power island. In some embodiments, the hardwareaccelerator 104 can be associated with a unique power island. In someembodiments, the memory fabric 106 can be associated with a unique powerisland. In some embodiments, the peripheral device 108 can be associatedwith a unique power island.

In some embodiments, the computing device 100 can invoke a power-upsequence by providing an enable signal to the power island. The enablesignal can subsequently close switches located between a power supplyvoltage and the power island, thereby providing the power supply voltageto the power island. This operation is sometimes referred to as powersupply gating.

FIG. 8 illustrates power supply gating of a power island in accordancewith some embodiments. FIG. 8 shows a power island 802, which mayinclude circuit blocks for processing input data, one or more switches804A-804B for providing a power supply voltage or a ground signal to thepower island 802, and an input register 806 for holding input data untilthe power island 802 is ready to process the input data. In someembodiments, the input register 806 is triggered to provide the inputdata to the power island 802 when the input register 806 receives avalid signal received from a valid signal generator 808, indicating thatthe power island 802 is ready to process the input data.

In some embodiments, the computing device 100 is configured to generatea valid signal indicating that the power supply voltage of the powerisland has reached an appropriate operating voltage. The valid signalcan indicate a time instance at which circuits in the power island canbe used to perform desired operations. The valid signal can be generatedby the valid signal generator 808.

The valid signal generator 808 could generate the valid signal using atimer. For example, the valid signal generator 808 can determine a timeinstance at which the enable signal is applied to the power-island, andwait a predetermined amount of time using a timer, and then generate thevalid signal. However, determining the predetermined amount of time atdesign time is difficult because the amount of time it takes to ramp upthe power supply voltage of a power island would be subject to process,voltage and temperature (PVT) variations. To address the PVT variations,the predetermined amount of time is often set conservatively (e.g., tobe sufficiently large) to accommodate worst-case PVT corners, which mayunnecessarily add latency to the power-up sequence.

To address these issues, in some embodiments, the valid signal generator808 is configured to generate the valid signal adaptively. Moreparticularly, the power island can be configured to generate the validsignal by adaptively delaying the enable signal provided to the powerisland.

FIG. 9 illustrates a valid signal generator in accordance with someembodiments. The valid signal generator 808 can include a plurality ofpower switches configured to provide a power supply voltage to logiccells coupled to the plurality of power switches. In some embodiments,the power switches can be a part of each logic cell. For example, thepower switches can include one or more P-channel devices in series withthe positive supply and/or one or more N-channel devices in series withthe negative supply (ground). These power switches can be distributedthroughout the logical block comprising the power-island. In FIG. 9, forsimplicity, the N and P-channel power switches are shown as a singlepower-switch block associated with each logic cell.

In some embodiments, the valid signal generator 808 can apply the enablesignal to the daisy chain of power switches and wait until the enablesignal reaches the end of the daisy chain of power switches. Once theenable signal reaches the end of the daisy chain of power switches, thenit is ensured that all logic cells in the power island are properlypowered on. Therefore, the valid signal generator 808 can use the enablesignal, delayed by the daisy chain of power switches, as the validsignal. This self-calibration mechanism can adaptively capture anyprocess-voltage-temperature (PVT) variations of the particular computingdevice. This way, the computing device need not unnecessarily wait along period of time for a power island to power-up; the computing devicecan wait only the amount of time needed to appropriately power-up thepower island.

In some embodiments, a power island can be always powered-on. In otherwords, a power island can be designed not to enter into a low-power modein which no power supply voltage is provided. Such a power island can bereferred to as an always-on power island.

In some embodiments, an always-on power-island can be used to monitorexternal signals. For example, the always-on power island can be used tomonitor signals from General-Purpose-Input-Output (GPIO) pins, externalinterfaces, and/or internal functional blocks such as a low frequencytimer or power-on reset. This way, the computing device 100 can analyzeexternal signals, determine whether one or more power islands need to bepowered up to respond to the external signals, and adaptively power-uponly the power-islands that are needed to respond to the externalsignals.

FIG. 10 illustrates an event signal monitoring mechanism in accordancewith some embodiments. FIG. 10 shows an always-on power island 802 and apower management module 110. The always-on power island 802 can includea power domain for the peripheral device 108. Since the always-on powerisland 802 does not enter into a low-power mode, the peripheral device108 in the always-on power island 802 can monitor signals that areasynchronous with a clock of the computing device 100. When theperipheral device 108 detects an event signal to which the computingdevice 100 should respond, the peripheral device 108 can alert the powermanagement module 110. In turn, the power management module 110 candetermine which one of the power islands in the computing device 100should be turned on. Subsequently, the power management module 110 cancause one or more of the power islands to be powered on.

In some embodiments, the peripheral device 108 can include a softwaredefined interface, whose functionality may be defined using software.More particularly, the peripheral devices 108 can include an interfaceprotocol emulation (IPE) module that is capable of emulating thefunctionality of standardized interface protocols, such as SPI, I2C,I2S, or any other suitable protocol. The software defined interface isbeneficial because the peripheral device 108 can maintain only a singlesoftware defined interface that can be programmed to accommodate aplurality of interface protocols, instead of maintaining a plurality ofinterfaces each dedicated to one particular interface protocol. Since asingle software defined interface can consume a lot less die areacompared to a plurality of dedicated interfaces, the single softwaredefined interface can drastically reduce the cost associated withinterfaces.

FIG. 11 shows a software defined interface in accordance with someembodiments. FIG. 11 shows a software defined interface that includes ageneric input/output (I/O) interface 1104, an IPE module 1106, and aninternal bus 1108 for a computing device 100. The generic input/outputinterface 1104 can include an interface for communicating with anexternal device, such as a sensor or a camera module.

The functionality of an I/O interface 1104 can be configured using anIPE module 1106. For example, when an IPE module 1106 determines thatthe I/O interface 1104 should operate as an I2C interface, then the IPEmodule 1106 can program the I/O interface 1104 to use the I2C interfaceprotocol for communication with the external device. In someembodiments, the IPE module 1106 can be programmed using software. TheIPE module 1106 can be programmed so that the IPE module 1106 canconfigure the I/O interface 1104 to implement standardized interfaceprotocols, such as SPI, I2C, I2S, or any other suitable standards.

FIG. 12 shows a detailed implementation of a software defined interfacein accordance with some embodiments. The software defined interface 1102can include a general-purpose input/output (GPIO) interface 1202 and itsregisters 1204. A host processor can control the operation of the GPIO1202 by configuring bits in the GPIO registers 1204. The GPIO 1202 cancontrol some of the pins in the I/O interface 1104 to communicate withexternal devices, such as an accelerometer, an ambient light sensor, oran audio sensor.

The software defined interface 1102 can also include an IPE module 1106and its registers 1206. A host processor can control the operation ofthe IPE module 1106 by configuring bits in the IPE registers 1206. TheIPE module 1106 can be configured to determine (1) an interface protocolto be implemented by the software defined interface 1102 and (2) I/Ointerface pins to be used to implement the interface protocol. Once theIPE module 1106 determines the I/O interface pins to be used toimplement the interface protocol, the IPE module 1106 can send a controlsignal to a multiplexer 1208 to multiplex the selected I/O interfacepins to the IPE module 1106. The IPE module 1106 can cause the I/Ointerface pins to emulate the interface protocol by causing the I/Ointerface pins to send control signals and data in accordance with theinterface protocol.

In some embodiments, the timer 1214 and/or the prescaler 1216 can beused to convert a high frequency reference clock (e.g., in the range ofhundreds of mega-hertz) to a low frequency clock (e.g., in the range ofhundreds of kilo-hertz) to provide an adequate clock signal to the IPE.In some embodiments, the frequency of the output clock from theprescaler 1216 can be multiplied by an integer value to emulate certaininterfaces. For example, when the output clock of the prescaler 1216 isoperating at 500 kHz, the frequency of the output clock from theprescaler 1216 can be multiplied by three to emulate I2C interfaces.This way, the 500 kHz clock can be used to operate the IPE logic and tosample the output registers connected to the I/O pins.

In some embodiments, the IPE module 1106 in the peripheral device 108can be configured to perform a bypass between input pins and output pinsof the I/O interface 1104, thereby emulating an input on one side of thecomputing device 100 and an output on the other side of the computingdevice 100 without actually powering up the processing units. Thisallows a first external device, such as an accelerometer coupled to thecomputing device 100 via I2C, to communicate with a second externaldevice, such as an application processor SoC, without waking up theprocessing units of the computing device 100.

The software defined interface 1102 can also include an event processor1210 and its registers 1212. The event processor 1210 can be configuredto receive external signals and detect any events to which the computingdevice 100 should respond. The functionality of the event processor 1210can be configured using EP registers 1212. In some embodiments, once theevent processor 1210 detects an event to respond to, the event processor1210 can determine the vector processors 102, hardware accelerators 104,and/or memory fabric 106 needed to respond to the event, and send apower-enable signal to the power island associated with the determinedvector processors 102, hardware accelerators 104, and/or memory fabric106.

FIG. 13 illustrates an event processor in accordance with someembodiments. As discussed above, the event processor 1210 maycommunicate with external devices and receive signals from the externaldevices. The signals can include audio samples, accelerometer values,ambient light sensor values, or any other inputs that can be providedvia a communication interface, such as a GPIO. The event processor 1210can be configured to compare the received signals to a particularconfiguration to recognize an event or a sequence of events. Once theevent processor 1210 recognizes an event or a sequence of events, theevent processor 1210 can cause one or more components in the computingdevice 100 to wake from a low power mode and commence operation.

In some embodiments, the event processor 1210 can include one or moreevent filters 1302A-1302N. An event filter 1302 is configured to receivean input signal from an interface 1104, and determine whether aparticular event has occurred. If the particular event has occurred, theevent filter 1302 can send a control signal and/or a power island enableto one of a plurality of power islands in the computing device 100.

FIG. 14 shows an implementation of an event filter in accordance withsome embodiments. The event filter 1302 can include a register 1402, acomparator 1404, and a Boolean operator 1406. The event filter 1302 canbe controlled via the event processor control registers 1212 and a timer1214.

The input registers 1402 can be configured to receive input signals fromone or more external devices and to provide the received input signals abank of comparators 1404. The comparators 1404 can be configured tosupport a wide range of input signal representations, including Boolean,integer, fixed-point, and floating-point representations.

Subsequently, the outputs from the comparators 1404 can be logicallycombined based on the timer value from the EP timer 1214 in order todetermine if a particular event or a sequence of events has happened. Insome cases, the particular event or the sequence of events is deemed tohave happened when a particular relationship between comparator outputspersists for a predetermined period of time. Once the event filter 1302determines that a particular event or a sequence of events has happened,the event filter 1302 can output control signals for controlling othercomponents in the computing device 100, such as a vector processor 102or a hardware accelerator 104, or external devices coupled to theperipheral device 108.

The event processor 1210 can be configured to detect an event in which auser starts using an electronic device. The event processor 1210 cansubsequently turn on components in the computing device 100 to respondto the start-up event. For instance, the event processor 1210 can beconfigured to detect a change in ambient light, which may indicate thatthe electronic device has been removed from a pocket. When the ambientlight remains at a high level for more than a few milliseconds, theevent processor 1210 can check the audio input to determine whetherthere is a change in input audio signals. When the event processor 1210detects a change in input audio signals, the event processor 1210 canenable a digital signal processor in the computing device 100 to detecta spoken command. This way, the event processor 1210 allows componentsin the computing device 100 to remain in a low-power mode and performoperations only when an event or a sequence of events has occurred.Therefore, the event processor 1210 can significantly reduce the averagestandby power of the computing device 100.

FIG. 15 shows a bypass mode of a peripheral device in accordance withsome embodiments. In FIG. 15, the computing device 100 can be in alow-power operation mode in which one or more power islands are in alow-power mode (e.g., no power supply voltage applied to the one or morepower islands). In this case, the IPE module 1106 can be configured toperform a bypass between input pins and output pins of the I/O interface1104, such as an input MIPI lane 1502 and an output MIPI lane 1504. Inthis example, the input MIPI lane 1502 is coupled to a camera module andthe output MIPI lane 1504 is coupled to an application processor.Therefore, the camera module can be coupled to the application processorwithout actually waking up the one or more power islands that are in alow-power mode.

In some embodiments, peripheral devices 108 for different interfaceprotocols can share physical pins (or pads) of the computing device 100.For example, the peripheral devices 108 can include a first interfacefor a first communication protocol and a second interface for a secondcommunication protocol. The first interface and the second interface canbe configured to time-multiplex the physical I/O pins so that the numberof I/O pins dedicated to the peripheral devices 108 can be reduced. Insome cases, the peripheral devices 108 can include a table that includesa mapping between signals in the first and second interfaces andphysical pins.

In applications where the computing device 100 is connected to a rangeof MIPI devices, such as cameras and displays, or to an applicationprocessor or other devices where the computing device 100 “appears” as acamera, the configuration of the computing device 100, in terms of thenumber of MIPI interface blocks and associated pins, may not be known atdesign time. For this reason, it is advantageous to connect a set ofMIPI I/O pins to a plurality of programmable MIPI I/O protocol controlblocks so that the number of MIPI inputs and outputs required to supporta particular MIPI use case can be configured at run-time via software.

FIG. 16 shows a programmable MIPI interface in accordance with someembodiments. The programmable MIPI interface 1600 can include a MIPImedia access control (MAC) protocol block 1602, a MIPI transmitter 1604,a MIPI receiver 1606, a multiplexer 1608 that is configured to channelsignals from either one of the MIPI transmitter 1604 or the MIPIreceiver 1606, a MIPI polarity switch 1610 that is configured to changethe polarity of the differential MIPI I/O pads 1612, and a bypassmultiplexer 1614 and a bypass buffer 1616 for performing a bypassbetween input pins and output pins of the I/O interface 1104 asillustrated with respect to FIG. 15.

In some embodiments, the MIPI MAC protocol block 1602 is designed tocontrol the operation of the MIPI transmitter 1604 and/or the MIPIreceiver 1606 so that the operation of the MIPI transmitter 1604 and/orthe MIPI receiver 1606 conforms with the MIPI protocol.

In some embodiments, the programmable MIPI interface 1600 can allow onlyone of the MIPI transmitter 1604 or the MIPI receiver 1606 tocommunicate via the MIPI I/O pad 1612 at a particular time instance. Forexample, the programmable MIPI interface 1600 can couple only one of theMIPI transmitter 1604 or the MIPI receiver 1606 with the MIPI I/O pad1612 via a multiplexer 1608. This way, to an external device, the MIPII/O pad 1612 can be considered a bi-directional MIPI interface.

In some embodiments, the programmable MIPI interface 1600 can use theMIPI polarity switch 1610 to reverse the polarity of the differentialMIPI I/O pads so that the polarity of the differential MIPI I/O pads canbe reversed at run time in order to achieve better impedance matching orto correct errors in external PCB design without rework. FIG. 17illustrates an application of a polarity reversal mechanism for aninput/output interface in accordance with some embodiments. While FIG.17 illustrates the application of the polarity reversal mechanism forMIPI I/O pads, the polarity reversal mechanism can be used in a varietyof other interfaces that use differential pair of signal lines.

In some embodiments, as described generally above with respect to FIG.15, the programmable MIPI interface 1600 can provide a low-power MIPIbypass mode by providing MIPI multiplexers 1614 and buffers 1616 thatallow MIPI I/O pads 1612 to be connected to outputs without requiringthe processing units of the computing device 100 to be powered up. Thisfeature is desirable in modes where multiple camera sensors areconnected to the computing device 100 to perform computer vision taskswhile in other use-cases the computing device 100 is not required andthe application processor performs still or video image capture usingthe same set of sensors. With the provision of internal MIPImultiplexers 1614, such uses cases can be supported via the internalbypass multiplexers 1614 rather than using external components andgreatly simplifies the cost and complexity of the PCB on which the chipsare combined.

In some embodiments, the memory fabric 106 can include cache memory thatis designed to exploit data locality, in terms of both the spatial andtemporal locality. When a computing device 100 is not coupled to anexternal memory device, then the memory fabric 106 can allow vectorprocessors 102 and hardware accelerators 104 to use the cache memory asa general memory device. In some embodiments, the cache memory can bepartitioned into sections so that each section is exclusively used byone of the vector processors or one of the hardware accelerators.

In some embodiments, the memory fabric 106 is configured to maintainstate information of the computing device 100 when the computing device100 is in a power saving mode. This way, when the computing device 100is switched on again, the computing device 100 can redistribute thestate information to appropriate devices so that the delay associatedwith the “wake-up” procedure can be reduced.

In some cases, the state information is maintained in cache memory. Insuch cases, the cache memory that stores the state information may bepowered on even when the computing device 100 enters a power savingmode. The state information can include binaries of application(s)loaded at boot-time or during run-time. The state information can alsoinclude configuration information such as register settings, operatingmode, pipeline configuration, and runtime environment settings loaded atboot-time and modified during run-time which would otherwise have to bestored in external non-volatile memory and retrieved in the event of apower-down to power-up sequence. The state information can also includedata such as image data, and values from other sensors. The stateinformation can also include the state of communications protocolsbetween the computing device 100 and other system components which wouldotherwise need to be stored and retrieved from external non-volatilememory in the event of a power-down to power-up sequence.

In some embodiments, the memory fabric 106 can include a hardware-basedmutual-exclusion (mutex) controller 206. FIG. 18 illustrates a memoryfabric having a hardware-based mutex controller in accordance with someembodiments. FIG. 18 shows a plurality of processing units 1802A-1802P,a memory fabric 106, and a mutex controller 206. A processing unit 1802can include a vector processor 102 or a hardware accelerator 104. Themutex controller 206 can include one or more independently addressablemutex elements that are configured to coordinate multitasking ofprocessing units 1802 that share a data element. More particularly, amutex element can be configured to lock a shared data element, stored inthe memory fabric 106 or other parts of the computing device 100, to afirst processing unit 1802A so that other processing units 1802P thatalso use the shared data element can wait until the first processingunit 1802A releases the shared data element. Because the mutexcontroller 206 resides within the memory fabric 106, the time to releaseor lock a shared resource is reduced when compared to using a shared busor other means.

Traditionally, when a mutex controller receives a request for anexclusive access to a shared resource, the mutex controller immediatelyresponds to the request, indicating whether the requesting processingunit can get an exclusive access to the shared resource. Therefore, ifthe requesting processing unit does not get an exclusive access, therequesting processing unit has to request the mutex controllercontinuously until the requesting processing unit receives the exclusiveaccess from the mutex controller. This can increase the traffic on thebus between the traditional mutex controller and processing units.

To address this issue, in some embodiments, when a processing unit 1802Asends an exclusive access request, requesting an exclusive access to ashared resource, the mutex controller 206 can monitor the status of therequest on its own. Once the mutex controller 206 determines that theprocessing unit 1802A is granted with the exclusive access, the mutexcontroller 206 can send an acknowledgment message to the processing unit1802A, indicating that the processing unit 1802A has the exclusiveaccess to the shared resource. This way, the processing unit 1802A isnot required to send the exclusive access request multiple times untilthe processing unit 1802A receives the exclusive access; the processingunit 1802A can send the exclusive access request only once and wait toreceive the exclusive access from the mutex controller 206. Thismessaging mechanism can reduce the communication load on the memoryfabric 106.

In some embodiments, the memory fabric 106 can include a flexible busarchitecture that provides communication between processing units.Frequently, an interface for communication between processing unitsincludes a buffer, such as a First-In-First-Out (FIFO). For example,when a first processing unit is ready to send a message to a secondprocessing unit, the first processing unit can send the message to abuffer that is assigned to the second processing unit. When the secondprocessing unit is ready to receive the message, the second processingunit can retrieve the message from the buffer.

However, buffers in traditional interfaces have a limited storagecapacity. Therefore, buffers in traditional interfaces are often limitedto storing control messages and could not accommodate large amount ofdata, such as image and video data. Furthermore, each buffer ispermanently assigned to one of the processing units. Therefore, while afirst buffer assigned to a first processing unit may be overflowing, asecond buffer assigned to a second processing unit may be empty. Thus,the capacity of buffers may not be fully utilized at the system level.

The memory fabric 106 addresses these shortcomings of traditionalinterfaces by increasing the capacity of buffers and by dynamicallyassigning buffers to processing units based on the real-time needs forcommunication. The memory fabric 106 provides a flexible mechanism forcreating, managing, and releasing buffers. The buffers can be createdfor the duration of a process, and can be released once the process iscompleted. The released buffer can be made available for otherapplications or processing units under a software program control.

FIG. 19 illustrates a dynamic assignment of buffers in accordance withsome embodiments. The memory fabric 106 can include a plurality ofbuffers 1902A-1902P, each of which may be exclusively assigned to one ofthe processing units, such as a vector processor or a hardwareaccelerator. In some cases, multiple buffers 1902 can be assigned to thesame processing unit.

In some embodiments, the plurality of buffers 1902 can be a part of arepository of buffers that can be partitioned and exclusively assignedto one of the processing units. The repository can comprise a memoryslice from the memory fabric 106. In some embodiments, each of theplurality of buffers 1902 may have the same capacity. In otherembodiments, one or more of the buffers 1902 may have a variablecapacity. For example, when a first processing unit 1802N attempts tosend a small number of control messages to a second processing unit1802C, the memory fabric 106 can assign a small buffer 1902C to thesecond processing unit 1802C so that the second processing unit 1802Ccan receive the small number of control messages. However, when thefirst processing unit 1802N attempts to send a large amount of videodata to the second processing unit 1802M, the memory fabric 106 canassign a buffer having a large capacity to the second processing unit1802M so that the second processing unit 1802M can receive the largeamount of video.

In some embodiments, one or more of the plurality of buffers 1902 can beassociated with particular applications, such as a communicationsinterface including USB, MIPI or Ethernet that can be foreseen at thedevice (system-on-chip) design time.

In some embodiments, the power management module 110 can be configuredto provide a different power supply voltage to logic circuits and memorydevices. FIG. 20 illustrates a power management mechanism that providesdifferent voltages to logic circuits memory devices in accordance withsome embodiments. A single power island 2002A can include a logiccircuit area 2004 and a memory area 2006. The power management module110 can be configured to provide a first voltage V₁ to the logic circuitarea 2004 and a second voltage V₂ to a memory area 2006. In someembodiments, the first voltage and the second voltage can be provided bya different power regulator. Therefore, the first voltage and the secondvoltage can be controlled independently.

In some embodiments, the logic circuit area 2004 and the memory area2006 can independently enter a low-power mode. For example, the powermanagement module 110 can use local switches 2008, 2010 to cut off thepower supply voltage to the logic circuit area 2004 and the memory area2006, respectively. In some embodiments, the power management module 110can use the global switch 2012 to cut off the power supply voltage tomemory areas 2006 in one or more power islands 2002A, . . . , 2002N.

In some embodiments, the memory fabric 106 can include a direct memoryaccess (DMA) engine. The DMA engine can maintain an operation list,which includes a double linked list of DMA data structures. Each DMAdata structure indicates a particular operation to be performed by theDMA engine. The DMA data structures are maintained in the order in whichthe DMA engine should perform operations associated with the DMA datastructures.

Because the operation list includes a double linked list of DMA datastructures, it takes a significant amount of time to remove a DMAoperation to the sequence of operations represented by the double linkedlist. In some embodiments, the DMA engine can address this issue bymaintaining a buffer that indicates whether a DMA data structure shouldbe executed. Each bit in the buffer can be considered an enable signalfor the associated DMA data structure.

FIG. 21 illustrates a DMA engine that implements a buffer-based DMA datastructure enable mechanism in accordance with some embodiments. The DMAengine includes an operation list 2102 that has a plurality of DMA datastructures 2104. The plurality of DMA data structures 2104 can becoupled to one another as a double linked list. The DMA engine alsoincludes an enable buffer 2106. The enable buffer 2106 can include aplurality of bits. The number of bits in the enable buffer 2106 can beidentical to the number of DMA data structures in the operation list2102. Each bit in the enable buffer 2106 can indicate whether a DMA datastructure associated with the bit is enabled. For example, when a firstbit in the buffer is a “1”, then the DMA engine can determine that thefirst DMA data structure is enabled and execute the first DMA datastructure. When a second bit in the buffer is a “0”, then the DMA enginecan determine that the second DMA data structure is enabled and notexecute the second DMA data structure. This way, the DMA engine canselectively execute a subset of the DMA data structures in the operationlist without actually removing DMA data structures from the operationlist. Since the DMA engine does not need to remove DMA data structures,the delay associated with disabling one or more DMA data structures canbe small.

In some embodiments, the parallel computing device 100 can reside in anelectronic device. FIG. 22 illustrates an electronic device thatincludes the computing device in accordance with some embodiments. Theelectronic device 2200 can include a processor 2202, memory 2204, one ormore interfaces 2206, and the computing device 100.

The electronic device 2200 can have memory 2204 such as a computerreadable medium, flash memory, a magnetic disk drive, an optical drive,a programmable read-only memory (PROM), and/or a read-only memory (ROM).The electronic device 2200 can be configured with one or more processors2202 that process instructions and run software that may be stored inmemory 2204. The processor 2202 can also communicate with the memory2204 and interfaces 2206 to communicate with other devices. Theprocessor 2202 can be any applicable processor such as asystem-on-a-chip that combines a CPU, an application processor, andflash memory, or a reduced instruction set computing (RISC) processor.

The memory 2204 can be a non-transitory computer readable medium, flashmemory, a magnetic disk drive, an optical drive, a programmableread-only memory (PROM), a read-only memory (ROM), or any other memoryor combination of memories. The software can run on a processor capableof executing computer instructions or computer code. The processor mightalso be implemented in hardware using an application specific integratedcircuit (ASIC), programmable logic array (PLA), field programmable gatearray (FPGA), or any other integrated circuit.

The interfaces 2206 can be implemented in hardware or software. Theinterfaces 2206 can be used to receive both data and control informationfrom the network as well as local sources, such as a remote control to atelevision. The electronic device can also provide a variety of userinterfaces such as a keyboard, a touch screen, a trackball, a touch pad,and/or a mouse. The electronic device may also include speakers and adisplay device in some embodiments.

In some embodiments, a processing unit, such as a vector processor 102and a hardware accelerator 104, in the computing device 100 can includean integrated chip capable of executing computer instructions orcomputer code. The processor might also be implemented in hardware usingan application specific integrated circuit (ASIC), programmable logicarray (PLA), field programmable gate array (FPGA), or any otherintegrated circuit.

In some embodiments, the computing device 100 can be implemented as asystem on chip (SOC). In other embodiments, one or more blocks in theparallel computing device can be implemented as a separate chip, and theparallel computing device can be packaged in a system in package (SIP).In some embodiments, the parallel computing device 400 can be used fordata processing applications. The data processing applications caninclude image processing applications and/or video processingapplications. The image processing applications can include an imageprocessing process, including an image filtering operation; the videoprocessing applications can include a video decoding operation, a videoencoding operation, a video analysis operation for detecting motion orobjects in videos. Additional applications of the present inventioninclude machine learning and classification based on sequence of images,objects or video and augmented reality applications including thosewhere a gaming application extracts geometry from multiple camera viewsincluding depth enabled cameras, and extracts features from the multipleviews from which wireframe geometry (for instance via a point-cloud) canbe extracted for subsequent vertex shading by a GPU.

The electronic device 2200 can include a mobile device, such as acellular phone. The mobile device can communicate with a plurality ofradio access networks using a plurality of access technologies and withwired communications networks. The mobile device can be a smart phoneoffering advanced capabilities such as word processing, web browsing,gaming, e-book capabilities, an operating system, and a full keyboard.The mobile device may run an operating system such as Symbian OS, iPhoneOS, RIM's Blackberry, Windows Mobile, Linux, Palm WebOS, and Android.The screen may be a touch screen that can be used to input data to themobile device and the screen can be used instead of the full keyboard.The mobile device may have the capability to run applications orcommunicate with applications that are provided by servers in thecommunications network. The mobile device can receive updates and otherinformation from these applications on the network.

The electronic device 2200 can also encompasses many other devices suchas televisions (TVs), video projectors, set-top boxes or set-top units,digital video recorders (DVR), computers, netbooks, laptops, tabletcomputers, and any other audio/visual equipment that can communicatewith a network. The electronic device can also keep global positioningcoordinates, profile information, or other location information in itsstack or memory.

It will be appreciated that whilst several different arrangements havebeen described herein, that the features of each may be advantageouslycombined together in a variety of forms to achieve advantage.

In the foregoing specification, the application has been described withreference to specific examples. It will, however, be evident thatvarious modifications and changes may be made therein without departingfrom the broader spirit and scope of the invention as set forth in theappended claims. For example, the connections may be any type ofconnection suitable to transfer signals from or to the respective nodes,units or devices, for example via intermediate devices. Accordingly,unless implied or stated otherwise the connections may for example bedirect connections or indirect connections.

It is to be understood that the architectures depicted herein are merelyexemplary, and that in fact many other architectures can be implementedwhich achieve the same functionality. In an abstract, but still definitesense, any arrangement of components to achieve the same functionalityis effectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermediate components. Likewise, any two componentsso associated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality.

Furthermore, those skilled in the art will recognize that boundariesbetween the functionality of the above described operations are merelyillustrative. The functionality of multiple operations may be combinedinto a single operation, and/or the functionality of a single operationmay be distributed in additional operations. Moreover, alternativeembodiments may include multiple instances of a particular operation,and the order of operations may be altered in various other embodiments.

However, other modifications, variations and alternatives are alsopossible. The specifications and drawings are, accordingly, to beregarded in an illustrative rather than in a restrictive sense.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word “comprising” does notexclude the presence of other elements or steps than those listed in aclaim. Furthermore, the terms “a” or “an,” as used herein, are definedas one or more than one. Also, the use of introductory phrases such as“at least one” and “one or more” in the claims should not be construedto imply that the introduction of another claim element by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim element to inventions containing only one suchelement, even when the same claim includes the introductory phrases “oneor more” or “at least one” and indefinite articles such as “a” or “an.”The same holds true for the use of definite articles. Unless statedotherwise, terms such as “first” and “second” are used to arbitrarilydistinguish between the elements such terms describe. Thus, these termsare not necessarily intended to indicate temporal or otherprioritization of such elements. The mere fact that certain measures arerecited in mutually different claims does not indicate that acombination of these measures cannot be used to advantage.

1. An electronic device comprising: a plurality of vector processors,wherein one of the plurality of vector processors is configured toexecute an instruction that operates on a first array of values; ahardware accelerator comprising a programmable datapath pipeline that isprogrammed using configuration information received from a softwaremodule, wherein the programmable datapath pipeline is configured toperform a filtering operation on a second array of values in accordancewith the configuration information; a memory fabric comprising aplurality of memory slices; and a host processor configured to cause thememory fabric to provide the first array of values to the one of theplurality of vector processors and to provide the second array of valuesto the hardware accelerator, thereby enabling the one of the pluralityof vector processors to process the first array of values in accordancewith the instruction and enabling the hardware accelerator to processthe second array of values in accordance with the configurationinformation.
 2. The electronic device of claim 1, wherein the hardwareaccelerator comprises: an output buffer for receiving a scan-line of animage processed by the programmable datapath pipeline; and a pipelinestall controller configured to stall an operation of the programmabledatapath pipeline when the output buffer is full.
 3. The electronicdevice of claim 1, wherein the hardware accelerator comprises aplurality of functional units that are chained together to perform thefiltering operation.
 4. The electronic device of claim 3, wherein anorder in which the plurality of functional units is chained together isdetermined using the configuration information received from thesoftware module.
 5. The electronic device of claim 4, wherein an outputof a first of the plurality of functional units is provided to a bufferin a memory fabric, and an input of a second of the plurality offunctional units is received from the buffer.
 6. The electronic deviceof claim 2, wherein the hardware accelerator further comprises a depthmap client that is configured to receive depth information that isindicative of a depth of an object represented by a pixel in thescan-line of the image.
 7. The electronic device of claim 6, wherein thehardware accelerator further comprises a depth map module that isconfigured to process the depth information to match a resolution of thedepth information to a resolution of the scan-line of the image.
 8. Theelectronic device of claim 7, wherein the depth map module is furtherconfigured to time-synchronize the depth information to the scan-line ofthe image.
 9. The electronic device of claim 1, wherein the memoryfabric comprises a mutex controller that is configured to monitor astatus of an exclusive access request requesting an exclusive access toa shared resource by one of the vector processors, and when the one ofthe vector processors receives an exclusive access to the sharedresource, send an acknowledgement message to the one of the vectorprocessors, indicating that the one of the vector processors has theexclusive access to the shared resource.
 10. The electronic device ofclaim 1, wherein the memory fabric comprises a plurality of buffers,wherein a first of the plurality of buffers is associated with a firstof the vector processors, and wherein a second of the vector processorsis configured to send data to the first of the vector processor bystoring the data in the first of the plurality of buffers.
 11. Theelectronic device of claim 10, wherein the memory fabric is configuredto dynamically modify a capacity of the first of the plurality ofbuffers based on an amount of data transferred to the first of thevector processors.
 12. The electronic device of claim 10, wherein thememory fabric is configured to dynamically associate two or more of theplurality of buffers to the first of the vector processors based on anamount of data transferred to the first of the vector processors. 13.The electronic device of claim 10, wherein the plurality of buffers is apart of one of the plurality of memory slices in the memory fabric. 14.The electronic device of claim 10, wherein the memory fabric isconfigured to store state information of one of the vector processorswhen the one of the vector processors enters a low-power mode.
 15. Theelectronic device of claim 14, wherein the state information is storedin a static random access memory in the memory fabric.
 16. Theelectronic device of claim 1, wherein the memory fabric comprises adirect memory access (DMA) controller, wherein the DMA controllercomprises an operation list indicating an order in which DMA operationsare to be performed.
 17. The electronic device of claim 16, wherein theDMA controller is configured to perform a subset of the DMA operationsin the operation list based on an enable buffer, wherein the enablebuffer includes a plurality of bits, wherein one of the plurality ofbits is associated with one of the DMA operations, and a value of theone of the plurality of bits is indicative of whether the one of the DMAoperations is to be performed by the DMA controller.
 18. A methodcomprising: providing, by a memory fabric comprising a plurality ofmemory slices, a first array of values to one of a plurality of vectorprocessors; providing, by the memory fabric, a second array of values toa hardware accelerator comprising a programmable datapath pipeline;executing, by one of the plurality of vector processors, an instructionthat operates on the first array of values; configuring the datapathpipeline in the hardware accelerator using configuration information;and performing, using the datapath pipeline in the hardware accelerator,a filtering operation on the second array of values in accordance withthe configuration information.
 19. The method of claim 18, furthercomprising: receiving, at an output buffer, a scan-line of an imageprocessed by the programmable datapath pipeline; and stalling, by apipeline stall controller, an operation of the programmable datapathpipeline when the output buffer is full.
 20. The method of claim 19,wherein the hardware accelerator comprises a plurality of functionalunits, and the method further comprises chaining the plurality offunctional units in accordance with the configuration information toperform the filtering operation.
 21. The method of claim 20, wherein theplurality of functional units comprises a first functional unit and asecond functional unit, and wherein chaining the plurality of functionalunits comprises an output of the first functional unit to an input ofthe second functional unit.
 22. The method of claim 19, furthercomprising: receiving depth information that is indicative of a depth ofan object represented by a pixel in the scan-line of the image; andsynchronizing the depth information to the scan-line of the image. 23.The method of claim 18, further comprising: monitoring, by a memorycontroller in the memory fabric, a status of an exclusive access requestrequesting an exclusive access to a shared resource by one of the vectorprocessors, and when the one of the vector processors receives anexclusive access to the shared resource, sending an acknowledgementmessage to the one of the vector processors, indicating that the one ofthe vector processors has the exclusive access to the shared resource.24. The method of claim 18, wherein the memory fabric comprises aplurality of buffers, wherein a first of the plurality of buffers isassociated with a first of the vector processors, and the method furthercomprises sending, by a second of the vector processors, data to thefirst of the vector processor by storing the data in the first of theplurality of buffers.
 25. The method of claim 24, further comprisingdynamically modifying a capacity of the first of the plurality ofbuffers based on an amount of data transferred to the first of thevector processors.
 26. The method of claim 24, further comprisingdynamically associating two or more of the plurality of buffers to thefirst of the vector processors based on an amount of data transferred tothe first of the vector processors.
 27. The method of claim 24, furthercomprising storing state information of one of the vector processors inthe memory fabric when the one of the vector processors enters alow-power mode.
 28. The method of claim 27, wherein the stateinformation is stored in a static random access memory in the memoryfabric.
 29. The method of claim 18, further comprising maintaining, at adirect memory access (DMA) controller, an operation list indicating anorder in which DMA operations are to be performed.
 30. The method ofclaim 29, further comprising performing a subset of the DMA operationsin the operation list based on an enable buffer, wherein the enablebuffer includes a plurality of bits, wherein one of the plurality ofbits is associated with one of the DMA operations, and a value of theone of the plurality of bits is indicative of whether the one of the DMAoperations is to be performed by the DMA controller.